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ABSTRACT 

This study examined the validity of an instrument 
designed to measure the effectiveness of faculty advisers to freshmen 
at a smail, private, liberal arts college (approximately 1,100 
students). The 19-question Advising Effectiveness Questionnaire (AEQ) 
was distributed to three successive freshman classes at the end of 
each academic year. Factor analysis indicated that the AEQ measured 
students' general level of satisfaction with the advising process and 
how they perceived the difficulty of their individual adjustment 
process independent of the advising process. The AEQ was also found 
to have a high level of reliability and strong face validity, which 
was further supported by the results of an outcome assessment study 
which used the AEQ to track improvement in advising. The survey 
instrument is appended. (MDM) 
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Evaluating Adviser Effectiveness 

George C. Fago, Ph.D. 
Ursinus College 

For both pragmatic and ethical reasons institutional change in 
educational programs should be evaluated in terms of outcome. For 
the sake of students themselves, for their instructors, and for 
administrators, when programmatic change is instituted, it is 
important to know what the nature and scope of the resulting 
effects are. Programmatic changes themselves, however, are 
typically variable from one institution to another because they 
arise from needs, problems, and goals that will also vary from one 
institution to the next. This variability in programmatic change 
has resulted in a lack of general instruments for outcome 
assessment; outcome assessment has tended to be idiosyncratic to 
each individual program for obvious reasons . 

Although programmatic change is necessarily particular to each 
institution, never- the- less programmatic change often involves the 
need to assess general factors that do transcend institutional 
considerations. As a case in point, while changes, in academic 
advising programs will typically involve a variety of changes and 
adjustments that differ across institutions, still there is a need 
to determine whether these changes have effected any change in the 
general effectiveness of the advising process. While there is a 
need for objective assessment instruments for these general 
factors, few if any such instruments have been developed. Indeed, 
there has been little discussion regarding what these general 
educational factors are, much less how they might be reliably and 
validly measured. For example, while there has been a continuing 
debate and discussion over the nature and merits of a 'liberal 
arts' educational program, there is little agreement as to what 
constitutes such a program or how it compares to other programs. 

The instrument to be described was developed in response to a 
mandate to assess the outcome of an institution-wide change in the 
handling of freshman advising. The older procedure forced students 
to declare a major at the beginning of their freshman year. 
Advising was then handled departmental ly and focused exclusively on 
course selection. In an effort to improve student retention and to 
facilitate the students' transition into the college setting, a new 
'freshman advising' program was developed. The newer program 
delayed the declaration of major until the end of the freshman 
year. Advising of freshmen was to be handled by designated 
'freshman advisors' who were trained and expected to take a much 
more holistic approach to the advising process. Advisers were 
expected to concern themselves not only with academic issues, but 
with personal and vocational issues as appropriate and needed. 
Because this represented a major departure from the established 
advising system, an outcome assessment was mandated. The Adviser 
Effectiveness Questionnaire was developed as the major tool for 
this assessment with the; explicit goal of assessing the degree of 
student satisfaction with the advising system. 
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B. Development of the Questionnaire 

The AEQ as it was developed reflected the ways in which the 
advising process itself had been redefined. Although academics, 
and in particular course selection, remained as a major area of 
advising emphasis, three additional areas of advising competence 
were identified: study skills, career/vocation, and personal 
counseling. Ideally the freshman advising process as envisioned 
should incorporate all four of these areas. Advisers first of all 
must be sensitive to student needs and interests and how best to 
account for them within a framework of institutional and 
departmental requirements for graduation, i.e. , academic advising. 
Secondly, for those many students who have yet to identify majors 
and/or vocational goals, advisers should be equipped to deal with 
low level career/vocational counseling. Third, advisors should 
also be ready to deal with problems involving the lack of study 
skills, since many students seemed to be deficient in this respect. 
Finally, advisors should also be prepared to deal with low level 
issues of a personal nature, e.g. , homesickness, roommate 
incompatibility, and performance anxiety. 

The AEQ contains nineteen items, ten of which were seen as 
directly related to the assessment of effectiveness and nine of 
which were included in an attempt to determine what problems in 
adjustment the students considered to be important and what 
problems they felt that advising should specifically address. Most 
of the items consisted of simple statements to which the students 
were to rate their degree of agreement on a Likert-type five point 
scale. The face validity of the items was transparent; e.g. , "I 
am satisfied with the way my advisor handled my questions regarding 
study skills." These statements were randomly divided into 
positive and negative statements in order to prevent response bias. 
In addition, two items were survey-type items designed to determine 
how frequently the students had met with their advisers and how 
accessible they felt their advisers to be. The last four items 
simply asked the students to rate the difficulty of adjustment to 
each of the four areas previously identified as possibly important 
to advising effectiveness. The actual scale items are reproduced 
in the appendix. In the following sections a psychometric 
evaluation of the instrument is presented utilizing data gathered 
as part of the outcome assessment of the advising program. 

METHODS 

Subjects : Three freshmen classes of students were assessed, each at 
the end of the freshman year. All were students at a small 
(enrollment approximately 1100 students) private, liberal arts 
college located in a suburban northeastern area of the country. 
Samples included both men and women, and in each case included 
approximately one third (125 respondents) of the freshman class. 
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Procedure : The AEQ was distributed by the freshmen advisers to the 
students who then returned them anonymously to the principle 
investigator. The distribution took place during the second half 
of the spring semester as part of the process of registration of 
major. Student participation was voluntary, but the importance of 
cooperation was stressed by the advisers. The first class assessed 
constituted a baseline condition, since it was handled by the older 
advising system. The two subsequent classes were advised under the 
newer system. 

RESULTS 

The validity of the instrument was assessed in two major ways. 
First, student responses to the questionnaire were analyzed for 
students under the older advising program compared to two classes 
under the newer program. An item by item analysis indicated that 
for eighteen of the nineteen items the two classes under the new 
system differed significantly from the baseline class (p<.05 or 
greater in all cases) although they did not differ between 
themselves. The differences were uniformly in a more positive 
direction. This seeming increase in student satisfaction as 
measured by the questionnaire was in agreement with a general 
faculty perception of a higher level of student satisfaction with 
advising, although this was not quantified. Thus student self- 
reported response to a program deliberately designed to improve the 
quality of advising, showed a uniform increase in reported levels 
of satisfaction with advising. This strongly suggests predictive 
validity of a sort. 

Secondly, in order to investigate the underlying construct 
validity of the AEQ the data drawn from a single class (N=114?) was 
analyzed by means of a factor analysis. Using the method of 
principal components analysis, a scree test indicated that a two 
factor solution was optimal. Equimax and varimax rotations 
resulted in identical results. The results of the factor analysis 
are presented in Table 1, which lists the £cale items in the order 
of the magnitude of their loadings on the twj factors respectively. 
Factor 1 consists of thirteen of the nineteen items; excluding all 
of the items where students rank the difficulty of adjustment, 
rating of overall level of happiness with the college, and 
agreement that more emphasis should be placed on academic 
counseling. These latter six items constituted the second factor. 



Insert Table 1 about here 



The factor analysis strongly suggests that the AEQ is 
measuring two independent factors. Factor 1 by inspection appears 
to reflect the students' global or general level of satisfaction 
with the advising process. Factor two seems to reflect how 
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students perceive the difficulty of their individual adjustment 
process independent of the advising process . 

The results of the factor analysis were then taken into 
consideration in the assessment of the reliability of the AEQ. 
Since the factor analysis was interpreted as indicating that a 
subset of thirteen scale items constituted a factor reflecting 
global satisfaction v:ith advising, instrument reliability was 
determined for these thirteen items. For the items in question 
Cronbach's alpha was computed after the items that had heavily 
loaded on the factor with a negative correlation had been reverse 
scored. The reliability coefficient estimated was r=0.8967 
indicating that the thirteen item scale was highly reliable. 

DISCUSSION 

The . psychometric assessment of the AEQ indicates that the 
instrument does have potential usefulness for the evaluation of 
undergraduate advising programs. It has a high level of 
reliability and strong face validity which is further borne out by 
the results of an outcome assessment study which used the 
instrument to track improvement in advising. It's usage with 
similar undergraduate programs is clearly indicated. Extensions to 
other programs non-residential students) is more problematic. 

The results of the factor analysis of the scale ai e of some 
additional interest. The original assumption that glided the 
construction of the scale was that undergraduate advising 
effectiveness might be a multidimensional construct; i.e. , advisers 
might well be evaluated on a number of independent dimensions. 
This did not seem to be the case, at least in the sample studied 
here. Rather the students seemed to view advising globally as a 
relationship which incorporates a number of aspects simultaneously, 
and to rate advising accordingly. Advisers seem to be perceived as 
'mentors' prepared to assist in a variety of ways, as opposed to 
'technicians- prepared to offer specific services. 

Factor 2 is also of interest. This factor consists of the 
four items where students rank order the difficulty of their 
personal adjustment in the various spheres, a rating of happiness, 
and. an item regarding the emphasis on academic advising. The fact 
that these items are independent of factor one suggests that 
students perceptions of their personal difficulties in adjustment 
do not color their assessment of the advising process. 
Interestingly enough, their happiness or lack thereof is correlated 
with factor 2 (difficulty of personal adjustments) and not with 
factor 1 (advising effectiveness) . Thus students satisfaction with 
college choice does not seem critically dependent upon the quality 
of advising (at least for the present sample) but rather depends 
upon the students own personal adjustment. 



6 



AEQ 



REFERENCES 

Chambliss, Catherine Anne & Fago, George (1987) . The Ursinus 
College Freshman Advising Program. NACADA Journal, 7, 20-22. 



ERIC 



AEQ 

Table 1: Rotated Factor Matrix 



LABEL 


ITEM 


FACTOR 1 


FACTOR 2 


COMFORT 


Comfortable talking with 


.83263 


- .03519 


OVERALL 


Overall satisfaction with 


.81624 


.01369 


STUDYSK 


Study skill advising satisfact 


. 79588 


.13915 


ACADEMIC 


Academic advising satisfaction 


. 74741 


.04093 


FREQUENC 


How frequent were meetings? 


- .74399 


.05254 


PERSONAL 


Personal advising satisfaction 


.71727 


- . 11940 


VOCATION 


Vocational Advising satisfact 


.66669 


- . 12145 


RELUCTAN 


Reluctant to discuss personal 


- .65990 


.07245 


ACCESS 


How accessible was advisor? 


- . 6464S 


- . 14490 


PREFSTUD 


Prefer students for advice 


- .51407 


.39030 


EMPVOCAT 


More emphasis on vocation 


.29106 


- .02152 


EMPSTUDY 


More emphasis on study skills 


- .25595 


.02548 


EMPPERS 


More emphasis on personal 


. 15109 . 


- .08556 










RANKADJ 


Difficulty of personal adj . 


.08288 


- .85771 


RANKACAD 


Difficulty of academic adj . 


.02945 


.55472 


HAPPY 


Happy with college choice 


- .01775 


.50786 


RANKVOC 


Difficulty of vocational adj . 


. 12449 


.35427 


EMPACAD 


More emphasis on academics 


.15539 


- .30074 


RANKSKIL 


Difficulty of study adj . 


- .20802 


.27839 



Factor Transformation Matrix 





FACTOR 1 


FACTOR 2 


FACTOR 1 


.98815 


- .15351 


FACTOR 2 


.15351 


.98815 
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APPENDIX 

Advising Effectiveness Questionnaire 

1 . How often during the past year did you see your advisor for 
advice or discussion? 

a. 1 time or less 

b. 2 to 5 times 

c. 6 to 10 times 

d. 11 times or more 

2. How accessible was your advisor to you? 

a. very accessible 

b. accessible 

c. neutral 

d. not very accessible 

e. not at all accessible 

The freshman year requires the typical student to make many 
adjustments. The general areas that frequently present 
problems in adjustment are listed below. Please rank them in 
terms of how difficult each area was in your own adjustment to 
college life. Rank each of the following areas on che answer 
sheet using the following key: 

"a" = the most difficult to adjust to 
"b" = the second most difficult to adjust to 
"c" = the third most difficult to adjust to 
"d" = the easiest to adjust to 

3. Study skills 

4 . Career 

5 . Academic 

6. Personal adjustment 

A number of statements dealing with various aspects of the 
advising process are presented below. For each statement you 
are to indicate the extent of your agreement or disagreement 
with the statement by checking the appropriate box with the 
following key: 

"a" = strongly agree 

"b" = agree 

"c" = neutral 

"d" = disagree 

"e" = strongly disagree 

7. I felt comfortable talking with my advisor. 

8. I am satisfied with the way my advisor handled my questions 
regarding career choices and decisions. 

9 . I am not satisfied with the way my advisor handled my 
questions regarding career choices and decisions. 
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10. I am satisfied with the way my advisor handled my questions 
regarding academic matters such as choice of courses, etc . 

11. I am satisfied with the way my advisor handled my questions 
regarding personal problems . 

12. Overall, I am not satisfied with the quality of the advising 
I received during my freshman year. 

13. In general, I am happy with my decision to come to this 
college. 

14. In student advising, more emphasis should be placed on helping 
students develop more adequate study skills. 

15. In student advising, less emphasis should be placed on helping 
students with career decisions. 

16. In student advising, less emphasis should be placed on helping 
students with academic problems. 

17. In student advising, more emphasis should be placed on helping 
students with personal problems . 

18. I am reluctant to talk about personal problems wiLh my 
advisor . 

19. I prefer to go to other students for advice and support. 
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